Addition of an ecFlow tool. by christinaholtNOAA · Pull Request #845 · ufs-community/uwtools

christinaholtNOAA · 2026-03-19T23:03:30Z

Synopsis

This is the basis for an ecFlow tool that translates YAML to an ecFlow definition.

Fixes Issue #847

Tests/docs/JSONSchema are all meant to be contributed separately.

In my working example, I'm testing with the following YAML:

ecflow:
  scheduler: "{{ platform.scheduler }}"
  suites_{{ ec.type }}:
    expand:
      type: [prod]
    family_primary:
      families_{{ ec.synop }}:
        expand:
          synop: !list '[{% for h in range(0, 23, 23) %}"{{ "%02d" % h }}",{% endfor %} ]'
        family_aigfs:
          vars:
            PROJ: AIGFS
            PACKAGEHOME: /scratch3/BMC/wrfruc/cholt/aigfs/aigfs/ecf
            EVAL: "YES"
          family_v1.0:
            family_forecast:
              task_jaigfs_prep:
                script: &script
                  account: myacct
                  pre_includes:
                    - head.h
                    - envir-1.h
                  post_includes:
                    - tail.h
                  manual: "Prepares AIGFS ICS from GFS."
                  execution: !dict "{{ prep.aigfs_ics.execution}}"
                  rundir: "{{ prep.aigfs_ics.rundir }}"
              task_jaigfs_forecast:
                trigger: jaigfs_prep == complete
                events: !list '[{% for h in range(0, 13, 6) %}"{{ "release_f%03d" % h }}",{% endfor %} ]'
                script:
                  <<: *script
                  manual: "Runs AIGFS forecast."
                  execution: !dict "{{ forecast.graphcast_model.execution}}"
                  rundir: "{{ forecast.graphcast_model.rundir }}"
              family_post:
                tasks_{{ ec.synop }}Z_post_f{{ ec.fhr }}:
                  expand:
                    fhr: !list '[{% for h in range(0, 13, 6) %}"{{ "%03d" % h }}",{% endfor %} ]'
                  trigger: "../jaigfs_forecast:release_f{{ ec.fhr }}"
                  script:
                    <<: *script
                    manual: "Runs AIGFS post for {{ ec.fhr }}."
                    execution: !dict "{{ post.aigfs_post.execution}}"
                    rundir: "{{ post.aigfs_post.rundir }}"
                  vars:
                    FHR: "{{ ec.fhr }}"

And the result is:

#5.15.2
suite prod
  family primary
    family 00
      family aigfs
        edit PROJ 'AIGFS'
        edit PACKAGEHOME '/scratch3/BMC/wrfruc/cholt/aigfs/aigfs/ecf'
        edit EVAL 'YES'
        family v1.0
          family forecast
            task jaigfs_prep
            task jaigfs_forecast
              trigger jaigfs_prep == complete
              event release_f000
              event release_f006
              event release_f012
            family post
              task 00Z_post_f000
                trigger ../jaigfs_forecast:release_f000
                edit FHR '000'
              task 00Z_post_f006
                trigger ../jaigfs_forecast:release_f006
                edit FHR '006'
              task 00Z_post_f012
                trigger ../jaigfs_forecast:release_f012
                edit FHR '012'
            endfamily
          endfamily
        endfamily
      endfamily
    endfamily
  endfamily
endsuite
# enddef

One key aspect that I haven't seen a in any prior solutions is the expand support. When included, as in the family_post set of tasks_*, multiple tasks will be created with the variables included in the expand section realized. This adds a looping mechanism to reduce duplication in the YAML definition.

The tool also writes all the task scripts, as defined by the task: script: blocks.

For example, prod/primary/00/aigfs/v1.0/forecast/forecast.ecf contains:

#SBATCH --account=myacct
#SBATCH --chdir={{ user.experiment_dir }}/{{ cycle.strftime("%Y%m%d%H") }}/forecast
#SBATCH --mem=150GB
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=128
#SBATCH --output={{ user.experiment_dir }}/{{ cycle.strftime("%Y%m%d%H") }}/forecast.out
#SBATCH --time=01:30:00
#SBATCH --verbose

model=%MODEL%

%include <head.h>
%include <envir-1.h>

source /scratch3/BMC/wrfruc/cholt/aigfs/etc/profile.d/conda.sh
conda activate aigfs

run graphcast
if [[ $? -ne 0 ]]; then
   ecflow_client --msg="***JOB $ECF_NAME ERROR RUNNING J-SCRIPT ***"
   ecflow_client --abort
   exit 1
fi

%include <tail.h>

%manual
Runs AIGFS forecast.
%end

I've been testing with a super simple script:

from pathlib import Path
from uwtools.ecflowtool import _ecFlowDef

config = Path("aigfs/ush/wflow.yaml")
e = _ecFlowDef(config)
print(e)
e.write_ecf_scripts(Path(".", "ecf"))
e.write_suite_definition(Path(".", "sd"))

Type

Bug fix (corrects a known issue)
Code maintenance (refactoring, etc. without behavior change)
Documentation
Enhancement (adds new functionality)
Tooling (CI, code-quality, packaging, revision-control, etc.)

Impact

This is a breaking change (changes existing functionality)
This is a non-breaking change (existing functionality continues to work as expected)

Checklist

I have added myself and any co-authors to the PR's Assignees list.
I have reviewed the documentation and have made any updates necessitated by this change.

christinaholtNOAA · 2026-03-23T15:29:35Z

Using Jinja2 expressions and dereferencing made things 100x easier! I added support for variables under Tasks, too, just to make sure I was getting the full benefit. Using an instance variable for these refs also means that a user can reference higher-level variables at any level, and we don't need to pass around that dict. I'm not sure that NCO typically wants to do that, but it is a pattern that's used in Rocoto, so it might come in handy for some users.

Here's the updated input YAML:

workflow:
  extern: []
  vars: []
  suites_{{ ec.type }}:
    repeat:
      type: [prod]
    family_primary:
      families_{{ ec.synop }}:
        repeat:
          synop: !list '[{% for h in range(0, 23, 23) %}"{{ "%02d" % h }}",{% endfor %} ]'
        family_aigfs:
          vars:
            PROJ: AIGFS
            PACKAGEHOME: /scratch3/BMC/wrfruc/cholt/aigfs/aigfs/ecf
            EVAL: YES
          family_v1.0:
            family_forecast:
              task_jaigfs_prep: {}
              task_jaigfs_forecast:
                trigger: jaigfs_prep == complete
                events: !list '[{% for h in range(0, 13, 6) %}"{{ "release_f%03d" % h }}",{% endfor %} ]'
              family_post:
                tasks_{{ ec.type }}_{{ ec.synop }}_post_f{{ ec.fhr }}:
                  repeat:
                    fhr: !list '[{% for h in range(0, 13, 6) %}"{{ "%03d" % h }}",{% endfor %} ]'
                  trigger: "../jaigfs_forecast:release_f{{ ec.fhr }}"
                  vars:
                    FHR: "{{ ec.fhr }}"

And the output suite definition where I've referenced synop and suite type in the task names:

#5.15.2
suite prod
  family primary
    family 00
      family aigfs
        family v1.0
          family forecast
            task jaigfs_prep
            task jaigfs_forecast
            family post
              task prod_00_post_f000
                edit FHR '000'
              task prod_00_post_f006
                edit FHR '006'
              task prod_00_post_f012
                edit FHR '012'
            endfamily
          endfamily
        endfamily
      endfamily
    endfamily
  endfamily
endsuite
# enddef

src/uwtools/ecflow.py

src/uwtools/ecflowtool.py

src/uwtools/ecflow.py

…into ecflow_tool

…tools into ecflow_tool

christinaholtNOAA · 2026-04-02T19:33:47Z

The test is failing due to 98% test coverage. Should I reduce the value, or should we override the failure when we merge?

Adding first pass at ecflow tool.

6c1a450

christinaholtNOAA self-assigned this Mar 20, 2026

Using Jinja2 + dereference for repeats.

ae39e8f

christinaholtNOAA commented Mar 23, 2026

View reviewed changes

src/uwtools/ecflow.py Outdated Show resolved Hide resolved

maddenp-cu reviewed Mar 23, 2026

View reviewed changes

src/uwtools/ecflowtool.py Outdated Show resolved Hide resolved

maddenp-cu reviewed Mar 25, 2026

View reviewed changes

src/uwtools/ecflow.py Show resolved Hide resolved

christinaholtNOAA added 4 commits March 25, 2026 17:12

Rename to ecflow for consistency with rocoto.

980485e

Updates to write ecf scripts.

1cd4b04

Linting and formatting.

584259d

Support for most attributes of Nodes.

602f755

christinaholtNOAA changed the base branch from main to ecflow_tool April 1, 2026 20:32

christinaholtNOAA and others added 6 commits April 1, 2026 14:33

Merge branch 'ecflow_tool' into ecflow_tool

2f92433

Cleanup the code a bit.

315ff67

Format and test.

de4379f

Merge branch 'ecflow_tool' of https://github.com/ufs-community/uwtools …

e3d89c1

…into ecflow_tool

Merge branch 'ecflow_tool' of https://github.com/christinaholtNOAA/uw…

d0c30e0

…tools into ecflow_tool

Add public methods for writing artifacts.

0702cd1

christinaholtNOAA marked this pull request as ready for review April 2, 2026 18:31

christinaholtNOAA requested a review from elcarpenterNOAA as a code owner April 2, 2026 18:31

christinaholtNOAA changed the title ~~ecFlow tool for early discussion~~ Addition of an ecFlow tool. Apr 2, 2026

christinaholtNOAA requested a review from AnnaSmoot-NOAA April 2, 2026 18:34

Put ecflow in the package recipe.

62f2baa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Addition of an ecFlow tool.#845

Addition of an ecFlow tool.#845
christinaholtNOAA wants to merge 13 commits intoufs-community:ecflow_toolfrom
christinaholtNOAA:ecflow_tool

christinaholtNOAA commented Mar 19, 2026 •

edited

Loading

Uh oh!

christinaholtNOAA commented Mar 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

christinaholtNOAA commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

christinaholtNOAA commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

christinaholtNOAA commented Mar 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

christinaholtNOAA commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

christinaholtNOAA commented Mar 19, 2026 •

edited

Loading